Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Option to have a Tsd as an index #314

Merged
merged 35 commits into from
Nov 6, 2024

Conversation

vigji
Copy link
Collaborator

@vigji vigji commented Jul 15, 2024

When working with Tsds, I might sometimes write, eg for filtering:

some_tsd = Tsd(...)
some_tsd.d[some_tsd.d > 500] = np.nan

I think it would be nice and not confusing for how the class works atm (very thorough feature equivalence with np.ndarray) having instead:

some_tsd = Tsd(...)
some_tsd[some_tsd > 500] = np.nan

Just by changing the setter and getter for the Tsd.

Would something like this make sense? This is just a draft for Tsd, it could be extended to other classes maybe although I never encountered the need.

@vigji vigji requested a review from gviejo as a code owner July 15, 2024 15:55
Copy link
Contributor

@gviejo gviejo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think there is a conflit with the previous PR

pynapple/core/time_series.py Show resolved Hide resolved
pynapple/core/time_series.py Outdated Show resolved Hide resolved
@gviejo
Copy link
Contributor

gviejo commented Jul 23, 2024

I think it's good and I would keep it only for time series.

@vigji
Copy link
Collaborator Author

vigji commented Jul 26, 2024

Everything should have been addressed now, let me know if there's anything else I should fix

@gviejo
Copy link
Contributor

gviejo commented Jul 29, 2024

That's great. I would still add one test for it.

@vigji
Copy link
Collaborator Author

vigji commented Aug 1, 2024

Excellent point, adding tests I actually realised I did not set the behavior for setitem in TsdFrame.

This should be now be good to go!

gviejo
gviejo previously approved these changes Aug 2, 2024
@gviejo gviejo dismissed their stale review August 2, 2024 16:39

missing for TsdTensor

@gviejo
Copy link
Contributor

gviejo commented Aug 2, 2024

It's missing for TsdTensor I think?

@gviejo gviejo self-requested a review August 2, 2024 16:40
gviejo
gviejo previously requested changes Aug 2, 2024
Copy link
Contributor

@gviejo gviejo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On line 732

    if isinstance(key, BaseTsd):
        key = key.d

@vigji
Copy link
Collaborator Author

vigji commented Aug 2, 2024

Oh sorry maybe I misunderstood one of your early messages - do you thing this should be implemented for Tsdtensors as well?

@vigji
Copy link
Collaborator Author

vigji commented Aug 2, 2024

Or you mean I should exclude tensors?

@gviejo
Copy link
Contributor

gviejo commented Aug 2, 2024

Yes I would implement it for TsdTensor too

@vigji
Copy link
Collaborator Author

vigji commented Sep 25, 2024

I'll come back to this soon if still unaddressed!

Since I am at it, I'll ask a slightly related but separate question: why is TsIndex not a Ts but a np.array? Is there some actual reason or it is just historical?

@gviejo
Copy link
Contributor

gviejo commented Sep 26, 2024

Basically TsIndex is like pd.Index in pandas. Ts is the user facing module. TsIndex is the non-user facing module which hold the timestamps and timestamps methods across all modules with time.

@vigji
Copy link
Collaborator Author

vigji commented Sep 27, 2024

This should now good to go!

I did a minor refactoring of test classes for readability, hopefully that's fine. Also, I'd love to do a separate PR for tests cleanup if that is good for you!

@vigji vigji requested a review from gviejo September 27, 2024 22:24
@vigji vigji dismissed gviejo’s stale review October 1, 2024 13:26

should fixed now

Copy link
Contributor

@gviejo gviejo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good a few changes only

pynapple/core/time_series.py Outdated Show resolved Hide resolved
pynapple/core/time_series.py Show resolved Hide resolved
pynapple/core/time_series.py Outdated Show resolved Hide resolved
vigji and others added 2 commits October 3, 2024 18:10
Copy link
Collaborator

@BalzaniEdoardo BalzaniEdoardo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Luigi! I did not go through the tests yet, the main code looks good to me, but I added a couple of suggestions;

pynapple/core/time_series.py Outdated Show resolved Hide resolved
pynapple/core/time_series.py Outdated Show resolved Hide resolved
pynapple/core/time_series.py Show resolved Hide resolved
@vigji
Copy link
Collaborator Author

vigji commented Oct 17, 2024

Let mw know if anything else is needed here!

(In the test refactoring, move test generated files to temporary folders would be very useful to avoid spurious commits)

@gviejo
Copy link
Contributor

gviejo commented Oct 17, 2024

Thanks I will look into it as soon as possible.

Copy link
Contributor

@gviejo gviejo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's almost good. Is it possible to add this for all time series :


            if not all(
                np.issubdtype(k.dtype, np.bool_) if isinstance(k, Tsd) else True
                for k in key
            ):
                raise ValueError(
                    "When indexing with a Tsd, it must contain boolean values"
                )

I should only be boolean indexing with another Tsd otherwise it doesn't make sense to index with integers because it means you are breaking the relationship between the time index and the corresponding values. Somehow it's similar to shuffling but it should be a proper method in the shuffling module.

@vigji
Copy link
Collaborator Author

vigji commented Nov 5, 2024

Ok, done it! I also change a bit the logic to bit more interpretable, and fixed the tests accordingly

# Conflicts:
#	pynapple/core/time_series.py
#	tests/test_time_series.py
@vigji
Copy link
Collaborator Author

vigji commented Nov 6, 2024

Should have fixed stuff and rebased on dev head. Would be great if we can merge so that I won't have to resolve more conflicts in the future!

@vigji vigji requested a review from gviejo November 6, 2024 00:07
@gviejo
Copy link
Contributor

gviejo commented Nov 6, 2024

Looks good thank you Luigi

@gviejo gviejo merged commit 6132f60 into pynapple-org:dev Nov 6, 2024
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants